Controversial California AI Law Moves Forward

The California Appropriations Committee passed the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (also known as SB-1047) on Thursday, the latest move in the saga of regulating Silicon Valley.

The state Assembly and Senate must still approve the bill before it becomes law.

What is SB-1047?

Known colloquially as California’s AI Act, and monitored closely across the country for possibly setting a precedent for state guidelines around generative AI, SB-1047 sets out several rules for AI developers:

• Create safety and security protocols for covered AI models.
• Ensure such models could be shut down completely.
• Prevent the distribution of models capable of what the act defines as “critical harm.”
• Retain an auditor to ensure compliance with the act.

In short, the bill provides a framework that aims to prevent generative AI models from causing large-scale damage to humanity, such as through nuclear war or bioweapons, or from causing over $500 million in losses through a cybersecurity event.

The act defines “covered models” as those using computing power greater than 10^26 integer or floating-point operations — the cost of which exceeds $100 million, during training.

Recent version of the act includes input from Anthropic

The version of the bill that passed on Thursday included some changes suggested by AI maker Anthropic and accepted by primary bill author Sen. Scott Wiener, D-Calif.

Anthropic successfully asked the state to remove some language from the bill which said companies in breach of the act could see legal action from the state’s attorney general. The newest version also removes the need for companies to disclose safety test results under threat of perjury. Instead, the developers will need to submit statements, which do not have the same legal weight.

Other changes include:

• The shift in wording from AI companies having to provide “reasonable assurance” of safety to “reasonable care.”
• An exception in which AI researchers who spend less than $10 million fine-tuning an open-source covered model are not considered developers of that model.

SEE: Anthropic and OpenAI have done their own digging into how generative AI creates content, including biased content.

The bill no longer calls for the creation of a Frontier Model Division, an agency to oversee the AI industry. Instead, a Board of Frontier Models focused on forward-looking safety guidance and audits will sit within the current Government Operations Agency.

While Anthropic contributed to the bill, other large organizations like Google and Meta have expressed disapproval. Andreessen Horowitz, a venture capital firm known as a16z that is behind many AI startups, has loudly opposed SB-1047.

Why is SB-1047 controversial?

Some representatives of industry and Congress say the act will restrict innovation and make it particularly difficult to work with open-source AI models. Among the bill’s critics was Hugging Face co-founder and CEO Clement Delangue, as noted by Fast Company.

An April study by the pro-regulation think tank Artificial Intelligence Policy Institute found that most Californians voted to support the bill as it stood at the time, with 70% agreeing “future powerful AI models may be used for dangerous purposes.”

Researchers Geoffrey Hinton and Yoshua Bengio, known as the “godfathers of AI” for their pioneering work on deep learning, publicly support the bill as well. The act will “protect the public,” Bengio wrote in an op-ed in Fortune on Aug. 15.

Eight of the 52 Congressional members from California signed a letter on Thursday saying the act would “create unnecessary risks for California’s economy with very little public safety benefit.” They argue that it is too early to create standardized evaluations for AI, as government agencies like NIST are still working on creating those standards.

They also suggest the definition of critical harm might be misleading. The bill has gone astray by focusing on large-scale disasters, such as nuclear weapons, while “largely ignoring demonstrable AI risks like misinformation, discrimination, nonconsensual deepfakes, environmental impacts, and workforce displacement,” the congress members claim.

SB-1047 includes specific protection for AI company whistleblowers under the California Whistleblower Protection Act.

Forrester Senior Analyst Alla Valente said legislators were right to focus on cyberattacks, such as the Change Healthcare incident in May, since these attacks are proven to cause serious harm. “With the use of generative AI, these attacks can be carried out more effectively and at a far greater scale which makes regulating AI something that all states will have to consider as part of how they protect and serve their residents,” she said.

The act brings up the challenges of balancing regulation and innovation

“We can advance both innovation and safety; the two are not mutually exclusive,” Wiener wrote in a public statement on Aug. 15. “While the amendments do not reflect 100% of the changes requested by Anthropic — a world leader on both innovation and safety — we accepted a number of very reasonable amendments proposed, and I believe we’ve addressed the core concerns expressed by Anthropic and many others in the industry.”

He noted that Congress is “gridlocked” on AI regulation, so “California must act to get ahead of the foreseeable risks presented by rapidly advancing AI while also fostering innovation.”

Next, the bill will need to be passed through the Assembly and Senate. If approved, the bill will then be considered by Gov. Gavin Newsom, likely in late August.

“Enterprise organizations are already having to contend with generative AI risks while navigating the enactment and enforcement of existing and emerging AI laws,” said Valente. “In the meantime, the growing AI litigation landscape is forcing organizations to prioritize AI governance at their organizations to ensure they address potential legal liabilities.  SB 1047 would create a guardrails and standards for AI products that would promote confidence in genAI-enabled products and could accelerate adoption.”